Batch Reinforcement Learning on the Industrial Benchmark: First Experiences

The Particle Swarm Optimization Policy (PSO-P) has been recently introduced and proven to produce remarkable results on interacting with academic reinforcement learning benchmarks in an off-policy, batch-based setting. To further investigate the properties and feasibility on real-world applications, this paper investigates PSO-P on the so-called Industrial Benchmark (IB), a novel reinforcement learning (RL) benchmark that aims at being realistic by including a variety of aspects found in industrial applications, like continuous state and action spaces, a high dimensional, partially observable state space, delayed effects, and complex stochasticity. The experimental results of PSO-P on IB are compared to results of closed-form control policies derived from the model-based Recurrent Control Neural Network (RCNN) and the model-free Neural Fitted Q-Iteration (NFQ). Experiments show that PSO-P is not only of interest for academic benchmarks, but also for real-world industrial applications, since it also yielded the best performing policy in our IB setting. Compared to other well established RL techniques, PSO-P produced outstanding results in performance and robustness, requiring only a relatively low amount of effort in finding adequate parameters or making complex design decisions

A Benchmark Environment Motivated by Industrial Control Problems

In the research area of reinforcement learning (RL), frequently novel and promising methods are developed and introduced to the RL community. However, although many researchers are keen to apply their methods on real-world problems, implementing such methods in real industry environments often is a frustrating and tedious process. Generally, academic research groups have only limited access to real industrial data and applications. For this reason, new methods are usually developed, evaluated and compared by using artificial software benchmarks. On one hand, these benchmarks are designed to provide interpretable RL training scenarios and detailed insight into the learning process of the method on hand. On the other hand, they usually do not share much similarity with industrial real-world applications. For this reason we used our industry experience to design a benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. In this paper we motivate and describe in detail the IB's dynamics and identify prototypic experimental settings that capture common situations in real-world industry control problems

A Benchmark Environment Motivated by Industrial Control Problems

In the research area of reinforcement learning (RL), frequently novel and promising methods are developed and introduced to the RL community. However, although many researchers are keen to apply their methods on real-world problems, implementing such methods in real industry environments often is a frustrating and tedious process. Generally, academic research groups have only limited access to real industrial data and applications. For this reason, new methods are usually developed, evaluated and compared by using artificial software benchmarks. On one hand, these benchmarks are designed to provide interpretable RL training scenarios and detailed insight into the learning process of the method on hand. On the other hand, they usually do not share much similarity with industrial real-world applications. For this reason we used our industry experience to design a benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. In this paper we motivate and describe in detail the IB's dynamics and identify prototypic experimental settings that capture common situations in real-world industry control problems

Multi-Jet Event Rates in Deep Inelastic Scattering and Determination of the Strong Coupling Constant

Jet event rates in deep inelastic ep scattering at HERA are investigated applying the modified JADE jet algorithm. The analysis uses data taken with the H1 detector in 1994 and 1995. The data are corrected for detector and hadronization effects and then compared with perturbative QCD predictions using next-to-leading order calculations. The strong coupling constant alpha_S(M_Z^2) is determined evaluating the jet event rates. Values of alpha_S(Q^2) are extracted in four different bins of the negative squared momentum transfer~\qq in the range from 40 GeV2 to 4000 GeV2. A combined fit of the renormalization group equation to these several alpha_S(Q^2) values results in alpha_S(M_Z^2) = 0.117+-0.003(stat)+0.009-0.013(syst)+0.006(jet algorithm).Comment: 17 pages, 4 figures, 3 tables, this version to appear in Eur. Phys. J.; it replaces first posted hep-ex/9807019 which had incorrect figure 4

Multiplicity Structure of the Hadronic Final State in Diffractive Deep-Inelastic Scattering at HERA

The multiplicity structure of the hadronic system X produced in deep-inelastic processes at HERA of the type ep -> eXY, where Y is a hadronic system with mass M_Y< 1.6 GeV and where the squared momentum transfer at the pY vertex, t, is limited to |t|<1 GeV^2, is studied as a function of the invariant mass M_X of the system X. Results are presented on multiplicity distributions and multiplicity moments, rapidity spectra and forward-backward correlations in the centre-of-mass system of X. The data are compared to results in e+e- annihilation, fixed-target lepton-nucleon collisions, hadro-produced diffractive final states and to non-diffractive hadron-hadron collisions. The comparison suggests a production mechanism of virtual photon dissociation which involves a mixture of partonic states and a significant gluon content. The data are well described by a model, based on a QCD-Regge analysis of the diffractive structure function, which assumes a large hard gluonic component of the colourless exchange at low Q^2. A model with soft colour interactions is also successful.Comment: 22 pages, 4 figures, submitted to Eur. Phys. J., error in first submission - omitted bibliograph

Differential (2+1) Jet Event Rates and Determination of alpha_s in Deep Inelastic Scattering at HERA

Events with a (2+1) jet topology in deep-inelastic scattering at HERA are studied in the kinematic range 200 < Q^2< 10,000 GeV^2. The rate of (2+1) jet events has been determined with the modified JADE jet algorithm as a function of the jet resolution parameter and is compared with the predictions of Monte Carlo models. In addition, the event rate is corrected for both hadronization and detector effects and is compared with next-to-leading order QCD calculations. A value of the strong coupling constant of alpha_s(M_Z^2)= 0.118+- 0.002 (stat.)^(+0.007)_(-0.008) (syst.)^(+0.007)_(-0.006) (theory) is extracted. The systematic error includes uncertainties in the calorimeter energy calibration, in the description of the data by current Monte Carlo models, and in the knowledge of the parton densities. The theoretical error is dominated by the renormalization scale ambiguity.Comment: 25 pages, 6 figures, 3 tables, submitted to Eur. Phys.

Hadron Production in Diffractive Deep-Inelastic Scattering

Characteristics of hadron production in diffractive deep-inelastic positron-proton scattering are studied using data collected in 1994 by the H1 experiment at HERA. The following distributions are measured in the centre-of-mass frame of the photon dissociation system: the hadronic energy flow, the Feynman-x (x_F) variable for charged particles, the squared transverse momentum of charged particles (p_T^{*2}), and the mean p_T^{*2} as a function of x_F. These distributions are compared with results in the gamma^* p centre-of-mass frame from inclusive deep-inelastic scattering in the fixed-target experiment EMC, and also with the predictions of several Monte Carlo calculations. The data are consistent with a picture in which the partonic structure of the diffractive exchange is dominated at low Q^2 by hard gluons.Comment: 16 pages, 6 figures, submitted to Phys. Lett.